Method of Improving Orientation and Color Balance of Digital Images Using Face Detection Information

ABSTRACT

A method of generating one or more new spatial and chromatic variation digital images uses an original digitally-acquired image which including a face or portions of a face. A group of pixels that correspond to a face within the original digitally-acquired image is identified. A portion of the original image is selected to include the group of pixels. Values of pixels of one or more new images based on the selected portion are automatically generated, or an option to generate them is provided, in a manner which always includes the face within the one or more new images. Such method may be implemented to automatically establish the correct orientation and color balance of an image. Such method can be implemented as an automated method or a semi automatic method to guide users in viewing, capturing or printing of images.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/637,664, filed Dec. 14, 2009; which is a continuation of U.S. patentapplication Ser. No. 11/765,899, filed Jun. 20, 2007, now U.S. Pat. No.7,912,245; which is a divisional to U.S. patent application Ser. No.10/608,772, filed Jun. 26, 2003, now U.S. Pat. No. 7,440,593; and whichis one of a series of contemporaneously-filed patent applicationsincluding application Ser. No. 10/608,810, entitled, “Digital ImageProcessing Using Face Detection Information”, application Ser. No.10/608,887, entitled, “Perfecting of Digital Image Capture ParametersWithin Acquisition Devices Using Face Detection”; application Ser. No.10/608,888, entitled, “Perfecting the Optics Within a Digital ImageAcquisition Device Using Face Detection”; application Ser. No.10/608,811, entitled, “Perfecting the Effect of Flash Within an ImageAcquisition Device Using Face Detection”; application Ser. No.10/608,772, “Method of Improving Orientation and Color Balance ofDigital Images Using Face Detection Information”; application Ser. No.10/608,784, entitled, “Modification of Viewing Parameters for DigitalImages Using Face Detection Information”; application Ser. No.10/608,766, entitled, “Digital Image Processing Composition Using FaceDetection Information”; application Ser. No. 10/608,819, entitled,“Digital Image Adjustable Compression and Resolution Using FaceDetection Information”; and application Ser. No. 10/608,824, entitled,“Perfecting of Digital Image Rendering Parameters Within RenderingDevices Using Face Detection”. This application is also related toapplication Ser. No. 11/024,046, entitled, “Detecting Orientation ofDigital Images Using Face Detection Information”; and PCT patentapplication no. PCT/US2006/021393, entitled, “Modification ofPost-Viewing Parameters for Digital Images Using Image Region or FeatureInformation”; all of which are herein incorporated by reference.

BACKGROUND

1. Field of the Invention

The invention relates to digital image processing, particularlyautomatic suggesting or processing of enhancements of a digital imageusing information gained from identifying and analyzing faces appearingwithin the image. The invention provides automated image processingmethods and tools for photographs taken and/or images detected, acquiredor captured in digital form or converted to digital form, or renderedfrom digital form to a soft or hard copy medium by using informationabout the faces in the photographs and/or images.

2. Description of the Related Art

Although well-known, the problem of face detection has not received agreat deal of attention from researchers. Most conventional techniquesconcentrate on face recognition, assuming that a region of an imagecontaining a single face has already been extracted and will be providedas an input. Such techniques are unable to detect faces against complexbackgrounds or when there are multiple occurrences in an image. For allof the image enhancement techniques introduced below and others as maybe described herein or understood by those skilled in the art, it isdesired to make use of the data obtained from face detection processesfor suggesting options for improving digital images or for automaticallyimproving or enhancing quality of digital images.

Yang et al., IEEE Transactions on Pattern Analysis and MachineIntelligence, Vol. 24, No. 1, pages 34-58, give a useful andcomprehensive review of face detection techniques January 2002. Theseauthors discuss various methods of face detection which may be dividedinto four main categories: (i) knowledge-based methods; (ii)feature-invariant approaches, including the identification of facialfeatures, texture and skin color; (iii) template matching methods, bothfixed and deformable and (iv) appearance based methods, includingeigenface techniques, statistical distribution based methods and neuralnetwork approaches. They also discuss a number of the main applicationsfor face detections technology. It is recognized in the presentinvention that none of this prior art describes or suggests usingdetection and knowledge of faces in images to create and/or use toolsfor the enhancement or correction of the images.

a. Faces as Subject Matter

Human faces may well be by far the most photographed subject matter forthe amateur and professional photographer. In addition, the human visualsystem is very sensitive to faces in terms of skin tone colors. Also, inexperiments performed by tracking the eye movement of the subjects, withan image that includes a human being, subjects tend to focus first andforemost on the face and in particular the eyes, and only later searchthe image around the figure. By default, when a picture includes a humanfigure and in particular a face, the face becomes the main object of theimage. Thus, many artists and art teachers emphasize the location of thehuman figure and the face in particular to be an important part of apleasing composition. For example, some teach to position faces aroundthe “Golden Ratio”, also known as the “divine proportion” in theRenaissance period, or PHI, φ-lines. Some famous artists whose workrepeatedly depict this composition are Leonardo Da-Vinci, Georges Seuratand Salvador Dali.

In addition, the faces themselves, not just the location of the faces inan image, have similar “divine proportion” characteristics. The headforms a golden rectangle with the eyes at its midpoint; the mouth andnose are each placed at golden sections of distance between the eyes andthe bottom on the chin etc. etc.

b. Color and Exposure of Faces

While the human visual system is tolerant to shifts in color balance,the human skin tone is one area where the tolerance is somewhat limitedand is accepted primarily only around the luminance axis, which is amain varying factor between skin tones of faces of people of differentraces or ethnic backgrounds. A knowledge of faces can provide animportant advantage in methods of suggesting or automatically correctingan overall color balance of an image, as well as providing pleasingimages after correction.

c. Auto Focus

Auto focusing is a popular feature among professional and amateurphotographers alike. There are various ways to determine a region offocus. Some cameras use a center-weighted approach, while others allowthe user to manually select the region. In most cases, it is theintention of the photographer to focus on the faces photographed,regardless of their location in the image. Other more sophisticatedtechniques include an attempt to guess the important regions of theimage by determining the exact location where the photographer's eye islooking. It is desired to provide advantageous auto focus techniqueswhich can focus on what is considered the important subject in the image

d. Fill-Flash

Another useful feature particularly for the amateur photographer isfill-flash mode. In this mode, objects close to the camera may receive aboost in their exposure using artificial light such as a flash, whilefar away objects which are not effected by the flash are exposed usingavailable light. It is desired to have an advantageous technique whichautomatically provides image enhancements or suggested options usingfill flash to add light to faces in the foreground which are in theshadow or shot with back light.

e. Orientation

The camera can be held horizontally or vertically when the picture istaken, creating what is referred to as a landscape mode or portraitmode, respectively. When viewing images, it is preferable to determineahead of time the orientation of the camera at acquisition, thuseliminating a step of rotating the image and automatically orienting theimage. The system may try to determine if the image was shothorizontally, which is also referred to as landscape format, where thewidth is larger than the height of an image, or vertically, alsoreferred to as portrait mode, where the height of the image is largerthan the width. Techniques may be used to determine an orientation of animage. Primarily these techniques include either recording the cameraorientation at an acquisition time using an in camera mechanicalindicator or attempting to analyze image content post-acquisition.In-camera methods, although providing precision, use additional hardwareand sometimes movable hardware components which can increase the priceof the camera and add a potential maintenance challenge. However,post-acquisition analysis may not generally provide sufficientprecision. Knowledge of location, size and orientation of faces in aphotograph, a computerized system can offer powerful automatic tools toenhance and correct such images or to provide options for enhancing andcorrecting images.

f. Color Correction

Automatic color correction can involve adding or removing a color castto or from an image. Such cast can be created for many reasons includingthe film or CCD being calibrated to one light source, such as daylight,while the lighting condition at the time of image detection may bedifferent, for example, cool-white fluorescent. In this example, animage can tend to have a greenish cast that it will be desired to beremoved. It is desired to have automatically generated or suggestedcolor correction techniques for use with digital image enhancementprocessing.

g. Cropping

Automatic cropping may be performed on an image to create a morepleasing composition of an image. It is desired to have automatic imageprocessing techniques for generating or suggesting more balanced imagecompositions using cropping.

h. Rendering

When an image is being rendered for printing or display, it undergoesoperation as color conversion, contrast enhancement, cropping and/orresizing to accommodate the physical characteristics of the renderingdevice. Such characteristic may be a limited color gamut, a restrictedaspect ratio, a restricted display orientation, fixed contrast ratio,etc. It is desired to have automatic image processing techniques forimproving the rendering of images.

i. Compression and Resolution

An image can be locally compressed in accordance with a preferredembodiment herein, so that specific regions may have a higher qualitycompression which involves a lower compression rate. It is desired tohave an advantageous technique for determining and/or selecting regionsof importance that may be maintained with low compression or highresolution compared with regions determined and/or selected to have lessimportance in the image.

SUMMARY OF THE INVENTION

In view of the above, a method of digital image processing using facedetection for achieving a desired spatial parameter is further providedincluding identifying a group of pixels that correspond to a face withina digital image, identifying one or more sub-groups of pixels thatcorrespond to one or more facial features of the face, determininginitial values of one or more parameters of pixels of the one or moresub-groups of pixels, determining an initial spatial parameter of theface within the digital image based on the initial values, anddetermining adjusted values of pixels within the digital image foradjusting the image based on a comparison of the initial and desiredspatial parameters.

The initial spatial parameter may include orientation. The values of thepixels may be automatically adjusted within the digital image to adjustthe initial spatial parameter approximately to the desired spatialparameter. An option may be automatically provided for adjusting thevalues of the pixels within the digital image to adjust the initialspatial parameter to the desired spatial parameter.

A method of digital image processing using face detection to achieve adesired orientation is also provided including identifying one or moregroups of pixels that correspond to a face within a digital image,identifying one or more sub-groups of pixels that correspond to one ormore facial features of the face, determining initial values of one ormore parameters of pixels of the one or more sub-groups of pixels,determining an initial orientation of the face within the digital imagebased on the initial values, and determining adjusted values of pixelswithin the digital image for adjusting the orientation to the desiredorientation.

Determining which of the sub-group of pixels belong to which of thegroup of face pixels may be performed. The determining of the initialvalues of one or more parameters of pixels may be calculated based onthe spatial orientation of the one or more sub-groups that correspond toone or more facial features. The spatial orientation of the one or moresub-groups that correspond to one or more facial features may becalculated based on an axis of an ellipse fit to the sub-group. Theadjusted values of pixels within the digital image may be rounded to aclosest multiple of 90 degrees. The initial values may be adjusted toadjusted values for re-orienting the image to an adjusted orientation.The one or more facial features may include an eye, two eyes, two eyesand a mouth, an eye, a mouth, hairline, ears, nostrils, nose bridge,eyebrows or a nose, or combinations thereof.

The method may include identifying one or more sub-groups of pixels thatcorrespond to one or more facial features of the face. Initial values ofone or more parameters of pixels of the one or more sub-groups of pixelsmay be determined. An initial spatial parameter of the face within thedigital image may be determined based on the initial values. The initialspatial parameter may include any of orientation, size and location.

When the spatial parameter is orientation, values of one or moreparameters of pixels may be adjusted for re-orienting the image to anadjusted orientation. The one or more facial features may include one ormore of an eye, a mouth, two eyes, a nose, an ear, and other facialfeatures. The one or more facial features may include two or morefeatures, and the initial orientation may be determined base on relativepositions of the features that are determined based on the initialvalues. A shape such as a triangle may be generated for example betweenthe two eyes and the center of the mouth, a golden rectangle asdescribed above, or more generically, a polygon having pointscorresponding to preferably three or more features as vertices or axis.

Initial values of one or more chromatic parameters, such as color andtone, of pixels of the digital image may be determined. The values ofone or more parameters may be automatically adjusted or an option toadjust the values to suggested values may be provided.

Each of the methods provided are preferably implemented within softwareand/or firmware either in the camera or with external processingequipment. The software may also be downloaded into the camera or imageprocessing equipment. In this sense, one or more processor readablestorage devices having processor readable code embodied thereon areprovided. The processor readable code programs one or more processors toperform any of the above or below described methods.

The processor may be located within a digital printing device, a digitalimage acquisition device, and/or a digital display device. Multipleimages may be displayed at once after the adjusting of the orientation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 a illustrates a preferred embodiment of the main workflow ofcorrecting images based on finding faces in the images.

FIG. 1 b illustrates a generic workflow of utilizing face information inan image to adjust image acquisition parameters in accordance with apreferred embodiment.

FIG. 1 c illustrates a generic workflow of utilizing face information ina single or a plurality of images to adjust the image renderingparameters prior to outputting the image in accordance with a preferredembodiment.

FIGS. 2 a-2 e illustrate image orientation based on orientation of facesin accordance with one or more preferred embodiments.

FIGS. 3 a-3 f illustrate an automatic composition and cropping of animage based on the location of the face in accordance with one or morepreferred embodiments.

FIGS. 4 a-4 g illustrate digital fill-flash in accordance with one ormore preferred embodiments.

FIG. 4 h describes an illustrative system in accordance with a preferredembodiment to determine in the camera as part of the acquisitionprocess, whether fill flash is needed, and of so, activate such flashwhen acquiring the image based on the exposure on the face

FIG. 5 illustrates the use of face-detection for generating dynamicslide shows, by applying automated and suggested zooming and panningfunctionality where the decision as to the center of the zoom is basedon the detection of faces in the image.

FIG. 6 describes an illustrative simulation of a viewfinder in a videocamera or a digital camera with video capability, with an automaticzooming and tracking of a face as part of the live acquisition in avideo camera, in accordance with a preferred embodiment.

FIGS. 7 a and 7 b illustrate an automatic focusing capability in thecamera as part of the acquisition process based on the detection of aface in accordance with one or more preferred embodiments.

FIG. 8 illustrates an adjustable compression rate based on the locationof faces in the image in accordance with a preferred embodiment.

INCORPORATION BY REFERENCE

What follows is a cite list of references each of which is, in additionto that which is described as background, the invention summary, theabstract, the brief description of the drawings and the drawingsthemselves, hereby incorporated by reference into the detaileddescription of the preferred embodiments below, as disclosingalternative embodiments of elements or features of the preferredembodiments not otherwise set forth in detail below. A single one or acombination of two or more of these references may be consulted toobtain a variation of the preferred embodiments described in thedetailed description herein:

U.S. Pat. Nos. RE33682, RE31370, 4,047,187, 4,317,991, 4,367,027,4,638,364, 5,291,234, 5,488,429, 5,638,136, 5,710,833, 5,724,456,5,781,650, 5,812,193, 5,818,975, 5,835,616, 5,870,138, 5,978,519,5,991,456, 6,097,470, 6,101,271, 6,128,397, 6,148,092, 6,151,073,6,188,777, 6,192,149, 6,249,315, 6,263,113, 6,268,939, 6,282,317,6,301,370, 6,332,033, 6,393,148, 6,404,900, 6,407,777, 6,421,468,6,438,264, 6,456,732, 6,459,436, 6,473,199, 6,501,857, 6,504,942,6,504,951, 6,516,154, and 6,526,161;

United States published patent applications no. 2003/0071908,2003/0052991, 2003/0025812, 2002/0172419, 2002/0114535, 2002/0105662,and 2001/0031142;

Japanese patent application no. JP5260360A2;

British patent application no. GB0031423.7; and

Yang et al., IEEE Transactions on Pattern Analysis and MachineIntelligence, Vol. 24, no. 1, pp 34-58 (January 2002).

Baluja & Rowley, “Neural Network-Based Face Detection,” IEEETransactions on Pattern Analysis and Machine Intelligence, Vol. 20, No.1, pages 23-28, January 1998

ILLUSTRATIVE DEFINITIONS

“Face Detection” involves the art of isolating and detecting faces in adigital image; Face Detection includes a process of determining whethera human face is present in an input image, and may include or ispreferably used in combination with determining a position and/or otherfeatures, properties, parameters or values of parameters of the facewithin the input image;

“Image-enhancement” or “image correction” involves the art of modifyinga digital image to improve its quality; such modifications may be“global” applied to the entire image, or “selective” when applieddifferently to different portions of the image. Some main categoriesnon-exhaustively include: (i) Contrast Normalization and ImageSharpening.

-   -   (ii) Image Crop, Zoom and Rotate.    -   (iii) Image Color Adjustment and Tone Scaling.    -   (iv) Exposure Adjustment and Digital Fill Flash applied to a        Digital Image.    -   (v) Brightness Adjustment with Color Space Matching; and        Auto-Gamma determination with Image Enhancement.    -   (vi) Input/Output device characterizations to determine        Automatic/Batch Image Enhancements.    -   (vii) In-Camera Image Enhancement    -   (viii) Face Based Image Enhancement

“Auto-focusing” involves the ability to automatically detect and bring aphotographed object into the focus field;

“Fill Flash” involves a method of combining available light, such as sunlight with another light source such as a camera flash unit in such amanner that the objects close to the camera, which may be in the shadow,will get additional exposure using the flash unit.

A “pixel” is a picture element or a basic unit of the composition of adigital image or any of the small discrete elements that togetherconstitute an image;

“Digitally-Captured Image” includes an image that is digitally locatedand held in a detector;

“Digitally-Acquired Image” includes an image that is digitally recordedin a permanent file and/or preserved in a more or less permanent digitalform; and

“Digitally-Detected Image”: an image comprising digitally detectedelectromagnetic waves.

“Digital Rendering Device”: A digital device that renders digitalencoded information such as pixels onto a different device. Most commonrendering techniques include the conversion of digital data into hardcopy such as printers, and in particular laser printers, ink jetprinters or thermal printers, or soft copy devices such as monitors,television, liquid crystal display, LEDs, OLED, etc.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments are described below including methods and devicesfor providing or suggesting options for automatic digital imageenhancements based on information relating to the location, position,focus, exposure or other parameter or values of parameters of faces inan image. Such parameters or values of parameter may include a spatialparameter. For example, an orientation of a face or faces within andigital image may be used to adjust or suggest an adjustment of anorientation of the entire image or of one or more faces within theimage. Color correction or enhancement may be automatically suggested orprovided for digital images based on color or tonal values of faces inthe image. Pleasing cropping of images may be suggested or providedbased on knowledge particularly of the location and/or size of faces inthe image.

A still image may be animated and used in a slide show by, e.g.,zooming, panning and/or rotating where the center point of an image iswithin a face or at least the face is included in all or substantiallyall of the images in the slide show. Selective compression, oralternatively selective resolution, or both, of images may be suggestedor automatically provided where one or more faces receive higher gradecompression and/or alternatively higher local resolution than otherportions of the image. A fill flash may be automatically digitallysimulated or suggested, or actually applied to an object, upon analysisof a luminance map of the image.

A camera may also be automatically focused prior to acquiring an imagebased on knowledge regarding the faces in an image.

A preferred embodiment includes an image processing application whetherimplemented in software or in firmware, as part of the image captureprocess, image rendering process, or as part of post processing. Thissystem receives images in digital form, where the images can betranslated into a grid representation including multiple pixels. Thisapplication detects and isolates the faces from the rest of the picture,and determines sizes and locations of the faces relative to otherportions of the image or the entire image. Orientations of the faces mayalso be determined. Based on information regarding detected faces,preferably separate modules of the system collect facial data andperform image enhancement operations based on the collected facial data.Such enhancements or corrections include automatic orientation of theimage, color correction and enhancement, digital fill flash simulationand dynamic compression.

In another embodiment, the information regarding the location and sizeof faces in an image assist in determining correct auto focus distanceand exposure in the camera. In a separate embodiment, such informationcan be processed in the camera as part of the post processing stage suchthat the saved image is already automatically corrected, enhanced and/oradjusted based on this information.

Advantages of the preferred embodiments include the ability toautomatically perform or suggest or assist in performing complex tasksthat may otherwise call for manual intervention and/or experimenting.Another advantage is that important regions, e.g., faces, of an imagemay be assigned, marked and/or mapped and then processing may beautomatically performed and/or suggested based on this informationrelating to important regions of the images. Automatic assistance may beprovided to a photographer in the post processing stage. Assistance maybe provided to the photographer in determining a focus and an exposurewhile taking a picture. Meta-data may be generated in the camera thatwould allow an image to be enhanced based on the face information.

Many advantageous techniques are provided in accordance with preferredand alternative embodiments set forth herein. For example, a method ofprocessing a digital image using face detection within said image toachieve one or more desired image processing parameters is provided. Agroup of pixels is identified that correspond to an image of a facewithin the digital image. Default values are determined of one or moreparameters of at least some portion of said digital image. Values of theone or more parameters are adjusted within the digitally-detected imagebased upon an analysis of said digital image including said image ofsaid face and said default values.

The digital image may be digitally-acquired and/or may bedigitally-captured. Decisions for processing the digital image based onsaid face detection, selecting one or more parameters and/or foradjusting values of one or more parameters within the digital image maybe automatically, semi-automatically or manually performed. Similarly,on the other end of the image processing workflow, the digital image maybe rendered from its binary display onto a print, or a electronicdisplay.

The one or more parameter may include orientation, color, tone, size,luminance, relative exposure, relative spatial location, tonereproduction, sharpness or focus or combinations thereof. The one ormore parameters may include a mask that defines one or more regionswhere the one or more parameters are valid. The mask may include acontinuous presentation of varying strength within different sub-regionsof said one or more regions. The one or more parameters may include thesame parameter differing in value based on said mask.

Two or more parameters may be concatenated into a single parameter. Thedigital image may be transformed based on values of the one or moreparameters. An operation list may be created for the digital image basedon values of the one or more parameters. The operation list may beembedded within the digital image or may be external to the digitalimage.

Values of orientation may be adjusted such that a rotation value for thedigital image is determined. Values of the color, tone, size, luminance,relative exposure may be adjusted including manipulating a color, tonal,size, luminance, fill-flash balance of the digital image, respectively.Values of relative spatial location may be adjusted including adjustinga spatial location of an image of a face relative to at least one otherregion of the digital image. Values of focus may be adjusted includingadjusting values of focus for enhancing a focus of the image of the facewithin the digital image.

One or more different degrees of simulated fill flash may be created bymanual, semi-automatic or automatic adjustment. The analysis of theimage of the face may include a comparison of an overall exposure to anexposure around the identified face. The exposure may be calculatedbased on a histogram. Digitally simulation of a fill flash may includeoptionally adjusting tone reproduction and/or locally adjustingsharpness. One or more objects estimated to be closer to the camera orof higher importance may be operated on in the simulated fill-flash.These objects determined to be closer to the camera or of higherimportance may include one or more identified faces. A fill flash or anoption for providing a suggested fill-flash may be automaticallyprovided. The method may be performed within a digital acquisitiondevice, a digital rendering device, or an external device or acombination thereof.

The face pixels may be identified, a false indication of another facewithin the image may be removed, and an indication of a face may beadded within the image, each manually by a user, or semi-automaticallyor automatically using image processing apparatus. The face pixelsidentifying may be automatically performed by an image processingapparatus, and a manual verification of a correct detection of at leastone face within the image may be provided.

A method of digital image processing using face detection to achieve adesired image parameter is further provided including identifying agroup of pixels that correspond to an image of a face within adigitally-detected image. Initial values of one or more parameters of atleast some of the pixels are determined. An initial parameter of thedigitally-detected image is determined based on the initial values.Values of the one or more parameters of pixels within thedigitally-detected image are automatically adjusted based upon acomparison of the initial parameter with the desired parameter or anoption for adjusting the values is automatically provided.

The digitally-detected image may include a digitally-acquired, renderedand/or digitally-captured image. The initial parameter of thedigitally-detected image may include an initial parameter of the faceimage. The one or more parameters may include any of orientation, color,tone, size, luminance, and focus. The method may be performed within adigital camera as part of a pre-acquisition stage, within a camera aspart of post processing of the captured image or within externalprocessing equipment. The method may be performed within a digitalrendering device such as a printer, or as a preparation for sending animage to an output device, such as in the print driver, which may belocated in the printer or on an external device such as the PC, as partof a preparation stage prior to displaying or printing the image. Anoption to manually remove a false indication of a face or to add anindication of a face within the image may be included. An option tomanually override, the automated suggestion of the system, whether ornot faces were detected, may be included.

The method may include identifying one or more sub-groups of pixels thatcorrespond to one or more facial features of the face. Initial values ofone or more parameters of pixels of the one or more sub-groups of pixelsmay be determined. An initial spatial parameter of the face within thedigital image may be determined based on the initial values. The initialspatial parameter may include any of orientation, size and location.

When the spatial parameter is orientation, values of one or moreparameters of pixels may be adjusted for re-orienting the image to anadjusted orientation. The one or more facial features may include one ormore of an eye, a mouth, two eyes, a nose, an ear, neck, shouldersand/or other facial or personal features, or other features associatedwith a person such as an article of clothing, furniture, transportation,outdoor environment (e.g., horizon, trees, water, etc.) or indoorenvironment (doorways, hallways, ceilings, floors, walls, etc.), whereinsuch features may be indicative of an orientation. The one or morefacial or other features may include two or more features, and theinitial orientation may be determined base on relative positions of thefeatures that are determined based on the initial values. A shape suchas a triangle may be generated for example between the two eyes and thecenter of the mouth, a golden rectangle as described above, or moregenerically, a polygon having points corresponding to preferably threeor more features as vertices or axis.

Initial values of one or more chromatic parameters, such as color andtone, of pixels of the digital image may be determined. The values ofone or more parameters may be automatically adjusted or an option toadjust the values to suggested values may be provided.

Within a digital acquisition device, a method of perfecting acquisitionparameters of a digital image as part of an image capture process usingface detection within said captured image to achieve one or more desiredimage acquisition parameters is provided. Default values are determinedof one or more image attributes of at least some portion of the digitalimage. Values of one or more camera acquisition parameters aredetermined. Groups of pixels are identified that correspond to an imageof a face within the digitally-captured image. Corresponding imageattributes to the groups of pixels are determined. One or more defaultimage attribute values are compared with one or more captured imageattribute values based upon analysis of the image of the face. Cameraacquisition parameters are then adjusted corresponding to adjusting theimage attribute values.

The method may be performed within any digital image capture device,which as, but not limited to digital still camera or digital videocamera. The one or more parameters may include overall exposure,relative exposure, orientation, color balance, white point, tonereproduction, size, or focus, or combinations thereof. The face pixelsidentifying may be automatically performed by an image processingapparatus, and the method may include manually removing one or more ofthe groups of pixels that correspond to an image of a face. Anautomatically detected face may be removed in response to falsedetection of regions as faces, or in response to a determination toconcentrate on less image faces or images faces that were manuallydetermined to be of higher subjective significance, than facesidentified in the identifying step. A face may be removed by increasinga sensitivity level of said face identifying step. The face removal maybe performed by an interactive visual method, and may use an imageacquisition built-in display.

The face pixels identifying may be performed with an image processingapparatus, and may include manually adding an indication of another facewithin the image. The image processing apparatus may receive a relativevalue as to a detection assurance or an estimated importance of thedetected regions. The relative value may be manually modified as to theestimated importance of the detected regions.

Within a digital camera, a method of digital image processing using facedetection for achieving a desired image parameter is further provided. Agroup of pixels is identified that correspond to a face within a digitalimage. First initial values of a parameter of pixels of the group ofpixels are determined, and second initial values of a parameter ofpixels other than pixels of the group of pixels are also determined. Thefirst and second initial values are compared. Adjusted values of theparameter are determined based on the comparing of the first and secondinitial values and on a comparison of the parameter corresponding to atleast one of the first and second initial values and the desired imageparameter.

Initial values of luminance of pixels of the group of pixelscorresponding to the face may be determined. Other initial values ofluminance of pixels other than the pixels corresponding to the face mayalso be determined. The values may then be compared, and properties ofaperture, shutter, sensitivity and a fill flash may be determined forproviding adjusted values corresponding to at least some of the initialvalues for generating an adjusted digital image. The pixelscorresponding to the face may be determined according to sub-groupscorresponding to one or more facial features.

Within a digital acquisition device with an adjustable optical systemhaving an auto focusing mechanism, a method of perfecting the auto focusmechanism of the adjustable optical system as part of an image captureprocess using face detection in the image capture process to achieve oneor more desired image acquisition parameters is provided. Groups ofpixels are identified that correspond to an image of a face within adigitally-captured image. Corresponding image attributes to the groupsof pixels are determined. Auto focus is perfected by performing the autofocus on the plural groups of pixels that correspond to the image of theface.

The auto focus may be initially performed on the entire image. themethod for auto-focusing the lens and the automatic adjustingautomatically adjusting one or more properties of the adjustable opticalsystem. A user may manually activate the camera to perform theperfecting of the auto focus. The face pixels identifying may beautomatically performed by an image processing apparatus, and one ormore of the groups of pixels detected as faces may be manually removedin response to false detection of one or more regions as one or morefaces, or in response to a determination to concentrate on less imagefaces than faces identified in the identifying step. The faces may beremoved by increasing a sensitivity level of the face identifying stepand/or by an interactive visual method. An image acquisition built-indisplay may be used. A weighted average on individual objects of thegroups may be used in the auto-focus process. The face identifying stepmay be automatically performed by an image processing apparatus whichreceives a relative value as to detection assurance. In this case, aweighted average may be calculated based on the relative values as tothe detection assurance. The face pixels identifying may beautomatically performed by an image processing apparatus which receivesa relative value as to an estimated importance of the detected regions.In this case, A weighted average may be calculated based on the relativevalues as to the estimated detection assurance. The estimated importanceof the detected regions of faces may involve an analysis of a parametersuch as size of the faces, location of the faces within the capturedimage, or relative exposure of the faces, or combinations thereof.

Within a digital camera having a lens system, a method of adjusting thecapture parameters of a digitally-detected image based on detection offaces within the image to achieve a desired image parameter is alsoprovided. The method may be used for auto-focusing the lens as part ofthe acquisition process. One or more parameters of pixels of the faceimage are determined. Values of the one or more parameters of the pixelsmay be automatically adjusted based upon a comparison of the initialparameter with the desired parameter. For example, one or moreproperties of the lens system may be automatically adjusted based on thevalues to adjust the focus, an indication to the region of focus or anadjustment option may be provided. The one or more parameters mayinclude a spatial parameter such as a size and/or a location of the facein the image.

Within a digital acquisition device with a built in flash unit, a methodof perfecting the exposure of an acquired digital image using facedetection in the acquired image is provided. Groups of pixels thatcorrespond to plural images of faces are identified within a digitallyacquired image, and corresponding image attributes to the group ofpixels are determined. An analysis is performed of the correspondingattributes of the groups of pixels. It is then determined to activatethe built-in flash unit based on the analysis. An intensity of thebuilt-in flash unit is determined based on the analysis.

An initial step of calculating image attributes may be performed on anentire acquired digital image and image attributes may be compared tothe image attributes of the group of pixels. The image attributes mayinclude exposure. The exposure may be calculated as a function of one ormore parameters including aperture, speed, gain, or relativesensitivity, or combinations thereof. The groups of pixels of faces maybe given a certain weight based on weight criteria. The weight criteriamay be calculated based on a distance of the groups of pixels to thecamera. The weight criteria may be calculated based on relative sizes ofthe groups of pixels.

A pre-flash may be performed based on the calculated flash intensity todetermine whether the analysis is accurate. A second analysis may beperformed based on the pre-flash.

One or more different degrees of simulated fill flash may be created bymanual, semi-automatic or automatic adjustment. The analysis of theimage of the face may include a comparison of an overall exposure to anexposure around the identified face. The exposure may be calculatedbased on a histogram. Digitally simulation of a fill flash may includeoptionally adjusting tone reproduction and/or locally adjustingsharpness. One or more objects estimated to be closer to the camera orof higher importance may be operated on in the simulated fill-flash.These objects determined to be closer to the camera or of higherimportance may include one or more identified faces. A fill flash or anoption for providing a suggested fill-flash may be automaticallyprovided.

Within a digital camera, a further method of digital image processingusing face detection for achieving a desired image parameter isprovided. A group of pixels is identified that correspond to a facewithin a digital image. First initial values are determined of aparameter of pixels of the group of pixels. Second initial values of aparameter are determined of pixels other than pixels of the group ofpixels. The first and second initial values are compared. Adjustedvalues of the parameter are determined based on the comparing of thefirst and second initial values and on a comparison of the parametercorresponding to at least one of the first and second initial values andthe desired image parameter.

The parameter may include luminance, and the method may further includeautomatically generating the adjusted digital image using the adjustedvalues. The method may also further include automatically providing anoption to generate the adjusted digital image using the adjusted values.The adjusted values of the luminance may be provided by a fill flash, orby a digitally-simulated fill flash.

Within a digital camera, a further method of digital image processingusing face detection to achieve a desired luminance contrast isprovided. A group of pixels is identified that correspond to a facewithin a digital image. First initial values of luminance of pixels ofthe group of pixels are determined. Second initial values of luminanceof pixels other than pixels of the group of pixels are also determined.The first and second initial values are compared to determine an initialluminance contrast. Properties of a fill flash are determined forproviding adjusted values of luminance for at least some of the pixelsof the digital image based on a comparison of the initial luminancecontrast and the desired luminance contrast.

Within a digital rendering device, a further method of digital imageprocessing using face detection for achieving a desired image renderingparameters is provided. A group of pixels is identified that correspondto a face within a digital image. First initial values are determined ofa parameter of pixels of the group of pixels. Second initial values of aparameter are determined of pixels other than pixels of the group ofpixels. The first and second initial values are compared. Adjustedvalues of the parameter are determined based on the comparing of thefirst and second initial values and on a comparison of the renderingparameter corresponding to at least one of the first and second initialvalues and the desired image rendering parameter.

The parameter may include luminance, and the method may further includeautomatically generating the adjusted digital image using the adjustedvalues. The method may also further include automatically providing anoption to generate the adjusted digital image using the adjusted values.The adjusted values of the luminance may be provided by changing the inkcoverage, the display luminance values, etc.

Within a digital rendering device, a further method of digital imageprocessing using face detection to achieve a desired contrast and colorbalance is provided. A group of pixels is identified that correspond toa face within a digital image. First initial values of contrast and/orcolor balance of pixels of the group of pixels are determined. Secondinitial values of contrast and/or color balance of pixels other thanpixels of the group of pixels are also determined. The first and secondinitial values are compared to determine an initial contrast and/orcolor balance. Such tool may compensate for the disparity between theinput or digitally acquired image and the output device. Suchdiscrepancies may arise due to a mismatching of color gamut, thephysical characteristic of the display, reflective or self luminance thelimited contrast, the effect of the surrounding environment, etc.

A method of generating one or more new digital images using an originaldigitally-acquired image including a face is further provided. A groupof pixels that correspond to a face within the originaldigitally-acquired image is identified. A portion of the original imageis selected to include the group of pixels. Values of pixels of one ormore new images based on the selected portion are automaticallygenerated, or an option to generate them is provided, in a manner whichalways includes the face within the one or more new images.

A transformation may be gradually displayed between the originaldigitally-acquired image and one or more new images. Parameters of saidtransformation may be adjusted between the original digitally-acquiredimage and one or more new images. Parameters of the transformationbetween the original digitally-acquired image and one or more new imagesmay be selected from a set of at least one or more criteria includingtiming or blending or a combination thereof. The blending may includedissolving, flying, swirling, appearing, flashing, or screening, orcombinations thereof.

Methods of generating slide shows that use an image including a face areprovided in accordance with the generation of one or more new images. Agroup of pixels is identified that correspond to a face within adigitally-acquired image. A zoom portion of the image including thegroup of pixels may be determined. The image may be automatically zoomedto generate a zoomed image including the face enlarged by the zooming,or an option to generate the zoomed image may be provided. A centerpoint of zooming in or out and an amount of zooming in or out may bedetermined after which another image may be automatically generatedincluding a zoomed version of the face, or an option to generate theimage including the zoomed version of the face may be provided. One ormore new images may be generated each including a new group of pixelscorresponding to the face, automatic panning may be provided using theone or more new images.

A method of generating one or more new digital images using an originaldigitally-acquired image including a face is further provided. One ormore groups of pixels may be identified that correspond to two or morefaces within the original digitally-acquired image. A portion of theoriginal image may be selected to include the group of pixels. Values ofpixels may be automatically generated of one or more new images based onthe selected portion in a manner which always includes at least one ofthe two or more faces within the one or more new images or a panningintermediate image between two of the faces of said two or moreidentified faces or a combination thereof.

Panning may be performed between the two or more identified faces. Thepanning may be from a first face to a second face of the two or moreidentified faces, and the second face may then be zoomed. The first facemay be de-zoomed prior to panning to the second face. The second facemay also be zoomed. The panning may include identifying a panningdirection parameter between two of the identified faces. The panning mayinclude sequencing along the identified panning direction between thetwo identified faces according to the identified panning directionparameter.

A method of digital image processing using face detection for achievinga desired spatial parameter is further provided including identifying agroup of pixels that correspond to a face within a digital image,identifying one or more sub-groups of pixels that correspond to one ormore facial features of the face, determining initial values of one ormore parameters of pixels of the one or more sub-groups of pixels,determining an initial spatial parameter of the face within the digitalimage based on the initial values, and determining adjusted values ofpixels within the digital image for adjusting the image based on acomparison of the initial and desired spatial parameters.

The initial spatial parameter may include orientation. The values of thepixels may be automatically adjusted within the digital image to adjustthe initial spatial parameter approximately to the desired spatialparameter. An option may be automatically provided for adjusting thevalues of the pixels within the digital image to adjust the initialspatial parameter to the desired spatial parameter.

A method of digital image processing using face detection to achieve adesired orientation is also provided including identifying one or moregroups of pixels that correspond to a face within a digital image,identifying one or more sub-groups of pixels that correspond to one ormore facial features of the face, determining initial values of one ormore parameters of pixels of the one or more sub-groups of pixels,determining an initial orientation of the face within the digital imagebased on the initial values, and determining adjusted values of pixelswithin the digital image for adjusting the orientation to the desiredorientation.

Determining which of the sub-group of pixels belong to which of thegroup of face pixels may be performed. The determining of the initialvalues of one or more parameters of pixels may be calculated based onthe spatial orientation of the one or more sub-groups that correspond toone or more facial features. The spatial orientation of the one or moresub-groups that correspond to one or more facial features may becalculated based on an axis of an ellipse fit to the sub-group. Theadjusted values of pixels within the digital image may be rounded to aclosest multiple of 90 degrees. The initial values may be adjusted toadjusted values for re-orienting the image to an adjusted orientation.The one or more facial features may include an eye, two eyes, two eyesand a mouth, an eye, a mouth, hairline, ears, nostrils, nose bridge,eyebrows neck as an extension of the face, or a nose, or combinationsthereof, or otherwise as described above.

The method may include identifying one or more sub-groups of pixels thatcorrespond to one or more facial features of the face. Initial values ofone or more parameters of pixels of the one or more sub-groups of pixelsmay be determined. An initial spatial parameter of the face within thedigital image may be determined based on the initial values. The initialspatial parameter may include any of orientation, size and location.

When the spatial parameter is orientation, values of one or moreparameters of pixels may be adjusted for re-orienting the image to anadjusted orientation. The one or more facial features may include one ormore of an eye, a mouth, two eyes, a nose, an ear, and other facialfeatures including the neck as the physical extension of the face. Theone or more facial features may include two or more features, and theinitial orientation may be determined base on relative positions of thefeatures that are determined based on the initial values. A shape suchas a triangle may be generated for example between the two eyes and thecenter of the mouth, a golden rectangle as described above, or moregenerically, a polygon having points corresponding to preferably threeor more features as vertices or axis.

Initial values of one or more chromatic parameters, such as color andtone, of pixels of the digital image may be determined. The values ofone or more parameters may be automatically adjusted or an option toadjust the values to suggested values may be provided.

A method of digital image processing using face detection is alsoprovided wherein a first group of pixels that correspond to a facewithin a digital image is identified, and a second group of pixels thatcorrespond to another feature within the digital image is identified. Are-compositioned image is determined including a new group of pixels forat least one of the face and the other feature. The other feature mayinclude a second face. The re-compositied image may be automaticallygenerated or an option to generate the re-compositioned image may beprovided. Values of one or more parameters of the first and secondgroups of pixels, and relative-adjusted values, may be determined forgenerating the re-compositioned image.

A method of compression of an image including a face is also providedalso including identifying a group of pixels that correspond to a facewithin a digitally-acquired image. A first compression portion of theimage including the group of pixels is determined. A second compressionportion of the image other than the group of pixels is also determined.The first compression portion may be automatically compressed withhigher-grade compression than the second compression portion to generatea compressed image including the face, or an option to provided thecompressed image including the different grade compressions may beprovided.

A method of determining the necessary resolution of an image including aface is also provided also including identifying a group of pixels thatcorrespond to a face within a digitally-acquired image. A firstresolution portion of the image including the group of pixels isdetermined. A second resolution portion of the image other than thegroup of pixels is also determined. The first resolution portion may beautomatically compressed with higher-resolution than the secondresolution portion to generate a rendered image including the face, oran option to provided the compressed image including the different graderesolution may be provided. Such method may be used to save on renderingtime or consumables. For example, such method may determine using moreink on the more important regions of an image, and less ink on regionsof less importance, this saving on the overall ink consumption whenprinting an image.

Each of the methods provided are preferably implemented within softwareand/or firmware either in the camera, the rendering device such asprinters or display, or with external processing equipment. The softwaremay also be downloaded into the camera or image processing equipment. Inthis sense, one or more processor readable storage devices havingprocessor readable code embodied thereon are provided. The processorreadable code programs one or more processors to perform any of theabove or below described methods.

FIG. 1 a illustrates a preferred embodiment. An image is opened by theapplication in block 102. The software then determines whether faces arein the picture as described in block 106. If no faces are detected, thesoftware ceases to operate on the image and exits, 110.

Alternatively, the software may also offer a manual mode, where theuser, in block 116 may inform the software of the existence of faces,and manually marks them in block 118. The manual selection may beactivated automatically if no faces are found, 116, or it may even beoptionally activated after the automatic stage to let the user, via someuser interface to either add more faces to the automatic selection 112or even 114, remove regions that are mistakenly 110 identified by theautomatic process 118 as faces. Additionally, the user may manuallyselect an option that invokes the process as defined in 106. This optionis useful for cases where the user may manually decide that the imagecan be enhanced or corrected based on the detection of the faces.Various ways that the faces may be marked, whether automatically ofmanually, whether in the camera or by the applications, and whether thecommand to seek the faces in the image is done manually orautomatically, are all included in preferred embodiments herein.

In an alternative embodiment, the face detection software may beactivated inside the camera as part of the acquisition process, asdescribed in Block 104. This embodiment is further depicted in FIG. 1 b.In this scenario, the face detection portion 106 may be implementeddifferently to support real time or near real time operation. Suchimplementation may include sub-sampling of the image, and weightedsampling to reduce the number of pixels on which the computations areperformed.

In an alternative embodiment, the face detection software may beactivated inside the rendering device as part of the output process, asdescribed in Block 103. This embodiment is further depicted in FIG. 1 c.In this scenario, the face detection portion 106 may be implementedeither within the rendering device, or within a en external driver tosuch device.

After the faces are tagged, or marked, whether manually as defined in106, or automatically, 118, the software is ready to operate on theimage based on the information generated by the face-detection stage.The tools can be implemented as part of the acquisition, as part of thepost-processing, or both.

Block 120 describes panning and zooming into the faces. This tool can bepart of the acquisition process to help track the faces and create apleasant composition, or as a post processing stage for either croppingan image or creating a slide show with the image, which includesmovement. This tool is further described in FIG. 6.

Block 130 depicts the automatic orientation of the image, a tool thatcan be implemented either in the camera as art of the acquisition postprocessing, or on a host software. This tool is further described inFIGS. 2 a-2 e.

Block 140 describes the way to color-correct the image based on the skintones of the faces. This tool can be part of the automatic colortransformations that occur in the camera when converting the image fromthe RAW sensor data form onto a known, e.g. RGB representation, or laterin the host, as part of an image enhancement software. The various imageenhancement operations may be global, affecting the entire image, suchas rotation, and/or may be selective based on local criteria. Forexample, in a selective color or exposure correction as defined in block140, a preferred embodiment includes corrections done to the entireimage, or only to the face regions in a spatially masked operation, orto specific exposure, which is a luminance masked operation. Note alsothat such masks may include varying strength, which correlates tovarying degrees of applying a correction. This allows a localenhancement to better blend into the image.

Block 150 describes the proposed composition such as cropping andzooming of an image to create a more pleasing composition. This tool,150 is different from the one described in block 120 where the faces areanchors for either tracking the subject or providing camera movementbased on the face location.

Block 160 describes the digital-fill-flash simulation which can be donein the camera or as a post processing stage. This tool is furtherdescribed in FIGS. 4 a-4 e. Alternatively to the digital fill flash,this tool may also be an actual flash sensor to determine if a fillflash is needed in the overall exposure as described in Block 170. Inthis case, after determining the overall exposure of the image, if thedetected faces in the image are in the shadow, a fill flash willautomatically be used. Note that the exact power of the fill flash,which should not necessarily be the maximum power of the flash, may becalculated based on the exposure difference between the overall imageand the faces. Such calculation is well known to the one skilled in theart and is based on a tradeoff between aperture, exposure time, gain andflash power.

This tool is further described in FIG. 4 e. Block 180 describes theability of the camera to focus on the faces. This can be used as apre-acquisition focusing tool in the camera, as further illustrated inFIG. 7.

Referring to FIG. 1 b, which describes a process of using face detectionto improve in camera acquisition parameters, as aforementioned in FIG. 1a, block 106. In this scenario, a camera is activated, 1000, for exampleby means of half pressing the shutter, turning on the camera, etc. Thecamera then goes through the normal pre-acquisition stage to determine,1004, the correct acquisition parameters such as aperture, shutterspeed, flash power, gain, color balance, white point, or focus. Inaddition, a default set of image attributes, particularly related topotential faces in the image, are loaded, 1002. Such attributes can bethe overall color balance, exposure, contrast, orientation etc.

An image is then digitally captured onto the sensor, 1010. Such actionmay be continuously updated, and may or may not include saving suchcaptured image into permanent storage.

An image-detection process, preferably a face detection process, isapplied to the captured image to seek faces in the image, 1020. If noimages are found, the process terminates, 1032. Alternatively, or inaddition to the automatic detection of 1030, the user can manuallyselect, 1034 detected faces, using some interactive user interfacemechanism, by utilizing, for example, a camera display. Alternatively,the process can be implemented without a visual user interface bychanging the sensitivity or threshold of the detection process.

When faces are detected, 1040, they are marked, and labeled. Detectingdefined in 1040 may be more than a binary process of selecting whether aface is detected or not, It may also be designed as part of a processwhere each of the faces is given a weight based on size of the faces,location within the frame, other parameters described herein, etc.,which define the importance of the face in relation to other facesdetected.

Alternatively, or in addition, the user can manually deselect regions,1044 that were wrongly false detected as faces. Such selection can bedue to the fact that a face was false detected or when the photographermay wish to concentrate on one of the faces as the main subject matterand not on other faces. Alternatively, 1046, the user may re-select, oremphasize one or more faces to indicate that these faces have a higherimportance in the calculation relative to other faces. This process asdefined in 1046, further defines the preferred identification process tobe a continuous value one as opposed to a binary one. The process can bedone utilizing a visual user interface or by adjusting the sensitivityof the detection process.

After the faces are correctly isolated, 1040, their attributes arecompared, 1050 to default values that were predefined in 1002. Suchcomparison will determine a potential transformation between the twoimages, in order to reach the same values. The transformation is thentranslated to the camera capture parameters, 1070, and the image, 1090is acquired.

A practical example is that if the captured face is too dark, theacquisition parameters may change to allow a longer exposure, or openthe aperture. Note that the image attributes are not necessarily onlyrelated to the face regions but can also be in relations to the overallexposure. As an exemplification, if the overall exposure is correct butthe faces are underexposed, the camera may shift into a fill-flash modeas subsequently illustrated in FIG. 4 a-4 f.

Referring to FIG. 1 c, which describes a process of using face detectionto improve output or rendering parameters, as aforementioned in FIG. 1a, block 103. In this scenario, a rendering device such as a printer ora display, herein referred to as the Device, activated, 1100. Suchactivation can be performed for example within a printer, oralternatively within a device connected to the printer such as a PC or acamera. The device then goes through the normal pre-rendering stage todetermine, 1104, the correct rendering parameters such as tonereproduction, color transformation profiles, gain, color balance, whitepoint and resolution. In addition, a default set of image attributes,particularly related to potential faces in the image, are loaded, 1102.Such attributes can be the overall color balance, exposure, contrast,orientation etc.

An image is then digitally downloaded onto the device, 1110. Animage-detection process, preferably a face detection process, is appliedto the downloaded image to seek faces in the image, 1120. If no imagesare found, the process terminates, 1132 and the device resumes itsnormal rendering process. Alternatively, or in addition to the automaticdetection of 1130, the user can manually select, 1134 detected faces,using some interactive user interface mechanism, by utilizing, forexample, a display on the device. Alternatively, the process can beimplemented without a visual user interface by changing the sensitivityor threshold of the detection process. When faces are detected, 1040,they are marked, and labeled. Detecting defined in 1140 may be more thana binary process of selecting whether a face is detected or not, It mayalso be designed as part of a process where each of the faces is given aweight based on size of the faces, location within the frame, otherparameters described herein, etc., which define the importance of theface in relation to other faces detected.

Alternatively, or in addition, the user can manually deselect regions,1144 that were wrongly false detected as faces. Such selection can bedue to the fact that a face was false detected or when the photographermay wish to concentrate on one of the faces as the main subject matterand not on other faces. Alternatively, 1146, the user may re-select, oremphasize one or more faces to indicate that these faces have a higherimportance in the calculation relative to other faces. This process asdefined in 1146, further defines the preferred identification process tobe a continuous value one as opposed to a binary one. The process can bedone utilizing a visual user interface or by adjusting the sensitivityof the detection process.

After the faces are correctly isolated, 1140, their attributes arecompared, 1150 to default values that were predefined in 1102. Suchcomparison will determine a potential transformation between the twoimages, in order to reach the same values. The transformation is thentranslated to the device rendering parameters, 1170, and the image, 1190is rendered. The process may include a plurality of images. In this case1180, the process repeats itself for each image prior to performing therendering process. A practical example is the creation of a thumbnail orcontact sheet whish is a collection of low resolution images, on asingle display instance.

A practical example is that if the face was too dark captured, therendering parameters may change the tone reproduction curve to lightenthe face. Note that the image attributes are not necessarily onlyrelated to the face regions but can also be in relations to the overalltone reproduction.

Referring to FIGS. 2 a-2 e, which describe the invention of automaticrotation of the image based on the location and orientation of faces, ashighlighted in FIG. 1 Block 130. An image of two faces is provided inFIG. 2 a. Note that the faces may not be identically oriented, and thatthe faces may be occluding.

The software in the face detection stage, including the functionality ofFIG. 1 a, blocks 108 and 118, will mark the two faces, of the mother andson as an estimation of an ellipse 210 and 220 respectively. Using knownmathematical means, such as the covariance matrix of the ellipse, thesoftware will determine the main axis of the two faces, 212 and 222respectively as well as the secondary axis 214 and 224. Even at thisstage, by merely comparing the sizes of the axis, the software mayassume that the image is oriented 90 degrees, in the case that thecamera hel helo in landscape mode, which is horizontal, or in portraitmode which is vertical or +90 degrees, aka clockwise, or −90 degrees akacounter clockwise. Alternatively, the application may also be utilizedfor any arbitrary rotation value. However this information may notsuffice to decide whether the image is rotated clockwise orcounter-clockwise.

FIG. 2 c describes the step of extracting the pertinent features of aface, which are usually highly detectable. Such objects may include theeyes, 214, 216 and 224, 226, and the lips, 218 and 228. The combinationof the two eyes and the center of the lips creates a triangle 230 whichcan be detected not only to determine the orientation of the face butalso the rotation of the face relative to a facial shot. Note that thereare other highly detectable portions of the image which can be labeledand used for orientation detection, such as the nostrils, the eyebrows,the hair line, nose bridge and the neck as the physical extension of theface etc. In this figure, the eyes and lips are provided as an exampleof such facial features Based on the location of the eyes if found, andthe mouth, the image may, e.g., need to be rotated in a counterclockwise direction.

Note that it may not be enough to just locate the different facialfeatures, but it may be necessary to compare such features to eachother. For example, the color of the eyes may be compared to ensure thatthe pair of eyes originate form the same person. Another example is thatin FIGS. 2-c and 2-d, if the software combined the mouth of 218 with theeyes of 226, 224, the orientation would have been determined asclockwise. In this case, the software detects the correct orientation bycomparing the relative size of the mouth and the eyes. The above methoddescribes means of determining the orientation of the image based on therelative location of the different facial objects. For example, it maybe desired that the two eyes should be horizontally situated, the noseline perpendicular to the eyes, the mouth under the nose etc.Alternatively, orientation may be determined based on the geometry ofthe facial components themselves. For example, it may be desired thatthe eyes are elongated horizontally, which means that when fitting anellipse on the eye, such as described in blocs 214 and 216, it may bedesired that the main axis should be horizontal. Similar with the lipswhich when fitted to an ellipse the main axis should be horizontal.Alternatively, the region around the face may also be considered. Inparticular, the neck and shoulders which are the only contiguous skintone connected to the head can be an indication of the orientation anddetection of the face.

FIG. 2-e illustrates the image as correctly oriented based on the facialfeatures as detected. In some cases not all faces will be oriented thesame way. In such cases, the software may decide on other criteria todetermine the orientation of the prominent face in the image. Suchdetermination of prominence can be based on the relevant size of thefaces, the exposure, or occlusion.

If a few criteria are tested, such as the relationship between differentfacial components and or the orientation of individual components, notall results will be conclusive to a single orientation. This can be dueto false detections, miscalculations, occluding portions of faces,including the neck and shoulders, or the variability between faces. Insuch cases, a statistical decision may be implemented to address thedifferent results and to determine the most likely orientation. Suchstatistical process may be finding the largest results (simple count),or more sophisticated ordering statistics such as correlation orprincipal component analysis, where the basis function will be theorientation angle. Alternatively or in addition, the user may manuallyselect the prominent face or the face to be oriented. The particularorientation of the selected or calculated prominent face may itself beautomatically determined, programmed, or manually determined by a user.

The process for determining the orientation of images can be implementedin a preferred embodiment as part of a digital display device.Alternatively, this process can be implemented as part of a digitalprinting device, or within a digital acquisition device.

The process can also be implemented as part of a display of multipleimages on the same page or screen such as in the display of acontact-sheet or a thumbnail view of images. In this case, the user mayapprove or reject the proposed orientation of the images individually orby selecting multiple images at once. In the case of a sequence ofimages, this invention may also determine the orientation of imagesbased on the information as approved by the user regarding previousimages.

FIGS. 3 a-3 f describe an illustrative process in which a proposedcomposition is offered based on the location of the face. As defined inFIG. 1 a blocks 108 and 118, the face 320 is detected as are one or morepertinent features, as illustrated in this case, the eyes 322 and 324.The location of the eyes are then calculated based on the horizontal,330 and vertical 340 location. In this case, the face is located at thecenter of the image horizontally and at the top quarter vertically asillustrated in FIG. 3-d.

Based on common rules of composition and aesthetics, e.g., a face in aclose up may be considered to be better positioned, as in FIG. 3-e ifthe eyes are at the ⅔rd line as depicted in 350, and ⅓ to the left or ⅓to the right as illustrated in 360. Other similar rules may be thelocation of the entire face and the location of various portions of theface such as the eyes and lips based on aesthetic criteria such as theapplying the golden-ratio for faces and various parts of the face withinan image.

FIG. 3 c introduces another aspect of face detection which may happenespecially in non-restrictive photography. The faces may not necessarilybe frontally aligned with the focal plane of the camera. In this figure,the object is looking to the side exposing partial frontal, or partialprofile of the face. In such cases, the software may elect to use, thecenter of the face, which in this case may align with the left eye ofthe subject. If the subject was in full frontal position, the softwaremay determine the center of the face to be around the nose bridge. Thecenter of the face may be determined to be at the center of a rectangle,ellipse or other shape generally determined to outline the face or atthe intersection of cross-hairs or otherwise as may be understood bythose skilled in the art (see, e.g., ellipse 210 of FIGS. 2 b-2 e,ellipse 320 of FIG. 3 b, ellipse 330 of FIG. 3 c, the cross-hairs 350,360 of FIG. 3 e).

Based on the knowledge of the face and its pertinent features such aseyes, lips nose and ears, the software can either automatically or via auser interface that would recommend the next action to the user, cropportions of the image to reach such composition. For this specificimage, the software will eliminate the bottom region 370 and the rightportion 380. The process of re-compositioning a picture is subjective.In such case this invention will act as guidance or assistance to theuser in determining the most pleasing option out of potentially a few.In such a case a plurality of proposed compositions can be displayed andoffered to the user and the user will select one of them.

In an alternative embodiment, the process of re-compositioning the imagecan be performed within the image acquisition device as part of theimage taking process, whether as a pre-capture, pre-acquisition or postacquisition stage. In this scenario the acquisition device may display aproposed re-compositioning of the image on its display. Suchre-compositioning may be displayed in the device viewfinder or displaysimilarly to FIG. 3 f, or alternatively as guidelines of cropping suchas lines 352 and 354. A user interface such will enable the user toselect form the original composed image, or the suggested one. Similarfunctionality can be offered as part of the post acquisition orotherwise referred to the playback mode.

In additional embodiments, the actual lines of aesthetics, for examplethe ⅓^(th) lines 350 and 350, may also be displayed to the use asassistance in determining the right composition. Referring to FIGS. 4a-4 f, the knowledge of the faces may assist the user in creating anautomatic effect that is otherwise created by a fill-flash. Fill-flashis a flash used where the main illumination is available light. In thiscase, the flash assists in opening up shadows in the image.Particularly, fill flash is used for images where the object in theforeground is in the shadow. Such instances occur for example when thesun is in front of the camera, thus casting a shadow on the object inthe foreground. In many cases the object includes people posing in frontof a background of landscape.

FIG. 4 a illustrates such image. The overall image is bright due to thereflection of the sun in the water. The individuals in the foregroundare therefore in the shadow.

A certain embodiment of calculating the overall exposure can be doneusing an exposure histogram. Those familiar in the art may decide onother means of determining exposure, any of which may be used inaccordance with an alternative embodiment. When looking at the histogramof the luminance of the image at FIG. 4-b, there are three distinctareas of exposure which correspond to various areas. The histogramdepicts the concentration of pixels, as defined by the Y-Axis 416, as afunction of the different gray levels as defined by the X-axis 418. Thehigher the pixel count for a specific gray level, the higher the numberas depicted on the y-axis.

Regions 410 are in the shadows which belong primarily to the mother. Themidtones in area 412 belong primarily to the shaded foreground water andthe baby. The highlights 414 are the water. However, not all shadows maybe in the foreground, and not all highlights may be in the background. Acorrection of the exposure based on the histogram may result in anunnatural correction.

When applying face detection, as depicted in FIG. 4-c, the histogram inFIG. 4-d may be substantially more clear. In this histogram, region 440depicts the faces which are in the shadow. Note that the actualselection of the faces, as illustrated in 4-c need not be a binary maskbut can be a gray scale mask where the boundaries are feathered orgradually changing. In addition, although somewhat similar in shape, theface region 440 may not be identical to the shadow region of the entireimage, as defined, e.g., in FIG. 4 b at area 410. By applying exposurecorrection to the face regions as illustrated in FIG. 4-e, such aspassing the image through a lookup table 4-f, the effect is similar tothe one of a fill flash that illuminated the foreground, but did notaffect the background. By taking advantage of the gradual feathered maskaround the face, such correction will not be accentuated and noticed.

FIG. 4 e can also be performed manually thus allowing the user to createa varying effect of simulated fill flash. Alternatively, the softwaremay present the user with a selection of corrections based on differenttone reproduction curves and different regions for the user to choosefrom.

Although exposure, or tone reproduction, may be the most preferredenhancement to simulate fill flash, other corrections may apply such assharpening of the selected region, contrast enhancement, of even colorcorrection. Additional advantageous corrections may be understood bythose familiar with the effect of physical strobes on photographedimages.

Alternatively, as described by the flow chart of FIG. 4 g, a similarmethod may be utilized in the pre-acquisition stage, to determine if afill flash is needed or not. The concept of using a fill flash is basedon the assumption that there are two types of light sources thatilluminate the image: an available external or ambient light source,which is controlled by the gain, shutter speed and aperture, and a flashwhich is only controlled by the flash power and affected by theaperture. By modifying the aperture vs. the shutter speed, the cameracan either enhance the effect of the flash or decrease it, whilemaintaining the overall exposure. When the user activates the camera, inblock 104, as defined in FIG. 1 a, the camera calculates the overallexposure, 482. Such calculation is known to one skilled in the art andcan be as sophisticated as needed. In block 108, the camera searched forthe existence of faces in the image. An exposure is then calculated tothe regions defined as belonging to the faces, 486. The disparitybetween the overall exposure as determined in 484 and the faces, 486 iscalculated. If the face regions are substantially darker than theoverall exposure 486, the camera will then activate the flash in a fillmode, 490, calculate the necessary flash power, aperture and shutterspeed, 492 and acquire the image 494 with the fill flash. Therelationship between the flash power, the aperture and the shutter speedare well formulated and known to one familiar in the art of photography.Examples of such calculations can be found in U.S. Pat. No. 6,151,073 toSteinberg et. al., hereby incorporated by reference.

Alternatively, in a different embodiment, 496, this algorithm may beused to simply determine the overall exposure based on the knowledge andthe exposure of the faces. The image will then be taken, 488, based onthe best exposure for the faces, as calculated in 496. Many cameras havea matrix type of exposure calculation where different regions receivedifferent weights as to the contribution for the total exposure. In suchcases, the camera can continue to implement the same exposure algorithmwith the exception that now, regions with faces in them will receive alarger weight in their importance towards such calculations.

FIG. 5 describes yet another valuable use of the knowledge of faces inimages. In this example, knowledge of the faces can help improve thequality of image presentation. An image, 510 is inserted into a slideshow software. The face is then detected as defined in FIG. 1 block 104,including the location of the important features of the face such as theeyes and the mouth.

The user can then choose between a few options such as: zoom into theface vs. zoom out of the face and the level of zoom for a tight close up520, a regular close up 520 or a medium close up as illustrated by thebounding box 540. The software will then automatically calculate thenecessary pan, tilt and zoom needed to smoothly and gradually switchbetween the beginning and the end state. In the case where more than oneface is found, the software can also create a pan and zoom combinationthat will begin at one face and end at the other. In a more genericmanner, the application can offer from within a selection of effectssuch as dissolve,

FIG. 6 illustrates similar functionality but inside the device. Acamera, whether still or video as illustrated by the viewfinder 610,when in auto track mode 600, can detect the faces in the image, and thenpropose a digital combination of zoom pan and tilt to move from the fullwide image 630 to a zoomed in image 640. Such indication may also showon the viewfinder prior to zooming, 632 as indication to the user, whichthe user can then decide in real time whether to activate the autozooming or not. This functionality can also be added to a tracking modewhere the camera continuously tracks the location of the face in theimage. In addition, the camera can also maintain the right exposure andfocus based on the face detection.

FIG. 7 a illustrates the ability to auto focus the camera based on thelocation of the faces in the image. Block 710 is a simulation of theimage as seen in the camera viewfinder. When implementing a centerweight style auto focus, 718, one can see that the image will focus onthe grass, 17 feet away, as depicted by the cross 712. However, asdescribed in this invention, if the camera in the pre-acquisition mode,104 detects the face, 714, and focuses on the face, rather thanarbitrarily on the center, the camera will then indicate to the userwhere the focus is, 722 and the lens will be adjusted to the distance tothe face, which in this example, as seen in 728, is 11 ft. vs. theoriginal 17 ft.

This process can be extended by one skilled in the art to support notonly a single face, but multiple faces, by applying some weightedaverage. Such average will depend on the disparity between the faces, indistances, and sizes.

FIG. 7 b presents the workflow of the process as illustrated via theviewfinder in FIG. 7-a. When the face-auto-focus mode is activated, 740,the camera continuously seeks for faces, 750. This operation inside thecamera is performed in real time and needs to be optimized as such. Ifno faces are detected 760, the camera will switch to an alternativefocusing mode, 762. If faces are detected, the camera will mark thesingle or multiple faces. Alternatively, the camera may display thelocation of the face 772, on the viewfinder or LCD. The user may thentake a picture, 790 where the faces are in focus.

Alternatively, the camera may shift automatically, via user request orthrough preference settings to a face-tracking mode 780. In this mode,the camera keeps track of the location of the face, and continuouslyadjusts the focus based on the location of the face.

In an alternative embodiment, the camera can search for the faces andmark them, similarly to the cross in FIG. 722. The photographer can thenlock the focus on the subject, for example by half pressing the shutter.Locking the focus on the subject differs form locking the focus, by thefact that if the subject then moves, the camera can still maintain thecorrect focus by modifying the focus on the selected object.

FIG. 8 describes the use of information about the location and size offaces to determine the relevant compression ratio of different segmentsof the image. Given an image 800, which is segmented into tiles usinghorizontal grid 830, and vertical grid 820. The tiles which include orpartially include face information are marked 850. Upon compression,regions of 850 may be compressed differently than the tiles of image 800outside of this region. The degree of compression may be predetermined,pre-adjusted by the user or determined as an interactive process. In thecase of multiple detected faces in an image, the user may also assigndifferent quality values, or compression rates based on the importanceof the faces in the image. Such importance may be determinedsubjectively using an interactive process, or objectively usingparameters such as the relative size of the face, exposure or locationof the face relative to other subjects in the image.

An alternative method of variable compression involves variableresolution of the image. Based on this, the method described withreference to FIG. 8 can also be utilized to create variable resolution,where facial regions which are preferably usually the important regionsof the image, and will be preferably maintained with higher overallresolution than other regions in the image. According to this method,referring to FIG. 8, the regions of the face as defined in block 850will be preferably maintained with higher resolution than regions in theimage 800 which are not part of 850.

An image can be locally compressed so that specific regions will have ahigher quality compression which equates to lower compression rate.Alternatively and/or correspondingly, specific regions of an image mayhave more or less information associated with them. The information canbe encoded in a frequency-based, or temporal-based method such as JPEGor Wavelet encoding. Alternatively, compression on the spatial domainmay also involve a change in the image resolution. Thus, localcompression may also be achieved by defining adjustable variableresolution of an image in specific areas. By doing so, selected ordetermined regions of importance may maintain low compression or highresolution compared with regions determined to have less importance ornon-selected regions in the image.

Face detection and face tracking technology, particularly for digitalimage processing applications according to preferred and alternativeembodiments set forth herein, are further advantageous in accordancewith various modifications of the systems and methods of the abovedescription as may be understood by those skilled in the art, as setforth in the references cited and incorporated by reference herein andas may be otherwise described below. For example, such technology may beused for identification of faces in video sequences, particularly whenthe detection is to be performed in real-time. Electronic componentcircuitry and/or software or firmware may be included in accordance withone embodiment for detecting flesh-tone regions in a video signal,identifying human faces within the regions and utilizing thisinformation to control exposure, gain settings, auto-focus and/or otherparameters for a video camera (see, e.g., U.S. Pat. Nos. 5,488,429 and5,638,136 to Kojima et al., each hereby incorporated by reference). Inanother embodiment, a luminance signal and/or a color difference signalmay be used to detect the flesh tone region in a video image and/or togenerate a detecting signal to indicate the presence of a flesh toneregion in the image. In a further embodiment, electronics and/orsoftware or firmware may detect a face in a video signal and substitutea “stored” facial image at the same location in the video signal, whichmay be useful, e.g., in the implementation of a low-bandwidth videophone(see, e.g., U.S. Pat. No. 5,870,138 to Smith et al., hereby incorporatedby reference).

In accordance with another embodiment, a human face may be locatedwithin an image which is suited to real-time tracking of a human face ina video sequence (see, e.g., U.S. Pat. Nos. 6,148,092 and 6,332,033 toQian, hereby incorporated by reference). An image may be providedincluding a plurality of pixels and wherein a transformation andfiltering of each pixel is performed to determine if a pixel has a colorassociated with human skin-tone. A statistical distribution of skintones in two distinct directions may be computed and the location of aface within the image may be calculated from these two distributions.

In another embodiment, electrical and/or software or firmware componentsmay be provided to track a human face in an image from a video sequencewhere there are multiple persons (see, e.g., U.S. Pat. No. 6,404,900also to Qian, hereby incorporated by reference). A projection histogramof the filtered image may be used for output of the location and/or sizeof tracked faces within the filtered image. A face-like region in animage may also be detected by applying information to an observertracking display of the auto-stereoscopic type (see, e.g., U.S. Pat. No.6,504,942 to Hong et al., incorporated by reference).

An apparatus according to another embodiment may be provided fordetection and recognition of specific features in an image using aneigenvector approach to face detection (see, e.g., U.S. Pat. No.5,710,833 to Moghaddam et al., incorporated by reference). Additionaleigenvectors may be used in addition to or alternatively to theprincipal eigenvector components, e.g., all eigenvectors may be used.The use of all eigenvectors may be intended to increase the accuracy ofthe apparatus to detect complex multi-featured objects.

Another approach may be based on object identification and recognitionwithin a video image using model graphs and/or bunch graphs that may beparticularly advantageous in recognizing a human face over a widevariety of pose angles (see, e.g., U.S. Pat. No. 6,301,370 to Steffenset al., incorporated by reference). A further approach may be based onobject identification, e.g., also using eigenvector techniques (see,e.g., U.S. Pat. No. 6,501,857 to Gotsman et al., incorporated byreference). This approach may use smooth weak vectors to producenear-zero matches, or alternatively, a system may employ strong vectorthresholds to detect matches. This technique may be advantageouslyapplied to face detection and recognition in complex backgrounds.

Another field of application for face detection and/or trackingtechniques, particularly for digital image processing in accordance withpreferred and alternative embodiments herein, is the extraction offacial features to allow the collection of biometric data and trackingof personnel, or the classification of customers based on age, sex andother categories which can be related to data determined from facialfeatures. Knowledge-based electronics and/or software or firmware may beused to provide automatic feature detection and age classification ofhuman faces in digital images (see, e.g., U.S. Pat. No. 5,781,650 toLobo & Kwon, hereby incorporated by reference). Face detection andfeature extraction may be based on templates (see U.S. Pat. No.5,835,616 also to Lobo & Kwon, incorporated by reference). A systemand/or method for biometrics-based facial feature extraction may beemployed using a combination of disparity mapping, edge detection andfiltering to determine co-ordinates for facial features in the region ofinterest (see, e.g., U.S. Pat. No. 6,526,161 to Yan, incorporated byreference). A method for the automatic detection and tracking ofpersonnel may utilize modules to track a users head or face (see, e.g.,U.S. Pat. No. 6,188,777, incorporated by reference). For example, adepth estimation module, a color segmentation module and/or a patternclassification module may be used. Data from each of these modules canbe combined to assist in the identification of a user and the system cantrack and respond to a user's head or face in real-time.

The preferred and alternative embodiments may be applied in the field ofdigital photography. For example, automatic determination of mainsubjects in photographic images may be performed (see, e.g., U.S. Pat.No. 6,282,317 to Luo et al., incorporated by reference). Regions ofarbitrary shape and size may be extracted from a digital image. Thesemay be grouped into larger segments corresponding to physically coherentobjects. A probabilistic reasoning engine may then estimate the regionwhich is most likely to be the main subject of the image.

Faces may be detected in complex visual scenes and/or in a neuralnetwork based face detection system, particularly for digital imageprocessing in accordance with preferred or alternative embodimentsherein (see, e.g., U.S. Pat. No. 6,128,397 to Baluja & Rowley; and“Neural Network-Based Face Detection,” IEEE Transactions on PatternAnalysis and Machine Intelligence, Vol. 20, No. 1, pages 23-28, January1998 by the same authors, each reference being hereby incorporated byreference. In addition, an image may be rotated prior to the applicationof the neural network analysis in order to optimize the success rate ofthe neural-network based detection (see, e.g., U.S. Pat. No. 6,128,397,incorporated by reference). This technique is particularly advantageouswhen faces are oriented vertically. Face detection in accordance withpreferred and alternative embodiments, and which are particularlyadvantageous when a complex background is involved, may use one or moreof skin color detection, spanning tree minimization and/or heuristicelimination of false positives (see, e.g., U.S. Pat. No. 6,263,113 toAbdel-Mottaleb et al., incorporated by reference).

A broad range of techniques may be employed in image manipulation and/orimage enhancement in accordance with preferred and alternativeembodiments, may involve automatic, semi-automatic and/or manualoperations, and are applicable to several fields of application. Some ofthe discussion that follows has been grouped into subcategories for easeof discussion, including (i) Contrast Normalization and ImageSharpening; (ii) Image Crop, Zoom and Rotate; (iii) Image ColorAdjustment and Tone Scaling; (iv) Exposure Adjustment and Digital FillFlash applied to a Digital Image; (v) Brightness Adjustment with ColorSpace Matching; and Auto-Gamma determination with Image Enhancement;(vi) Input/Output device characterizations to determine Automatic/BatchImage Enhancements; (vii) In-Camera Image Enhancement; and (viii) FaceBased Image Enhancement.

(i) Contrast Normalization and Image Sharpening

This field is relates to adjusting a digital image, after capture, toimprove the image sharpness, contrast and/or potentially simulate animproved focus on the main subject. An image may be sharpened bytransforming the image representation into a frequency-domainrepresentation and by selectively applying scaling factors to certainfrequency domain characteristics of an image (see, e.g., U.S. Pat. No.6,421,468 to Ratnakar et al., incorporated by reference). The modifiedfrequency domain representation may be back-transformed into the spatialdomain and provide a sharpened version of the original image. This imagesharpening effect may be applied to the entire image (see particularlyRatnakar et al., above). Image sharpening may also be appliedselectively to particular spatial regions within an image in accordancewith an embodiment herein.

Automatic contrast enhancement of an image may be provided by increasingthe dynamic range of the tone levels within an image without causingsubstantial distortion or shifts to the color map of the image (see,e.g., U.S. Pat. No. 6,393,148 to Bhaskar, incorporated by reference).This enhancement may be applied to the entire image or selectively andadvantageously to a one or more particular spatial regions within theimage. In addition, correction for the entire image may be selectivelyderived from characteristics of a particular spatial region within theimage, such as a human face region.

A digital photo-finishing system may include image processing to providescene balance, image sharpening and/or contrast normalization (see,e.g., U.S. Pat. No. 6,097,470 to Buhr et al., incorporated byreference). Algorithms may be optimized to a print medium and applied tothe entire image.

(ii) Crop, Zoom and Rotate a Digital Image

The selection of a portion of a digital image with an improvedcompositional form over the original image represents a form of imageenhancement by “cropping”. A similar technique involves selecting asub-region of the original image and enhancing the resolution of thissub-region by interpolating between the pixels. This represents a formof digital zooming of the image and can provide an improvement on theoriginal image if correctly applied. A third means of spatially alteringthe image is to change the image orientation by rotating the image. Thismay be, e.g., a straight-forward 90° or 270° rotation to change theimage aspect from landscape to portrait or vice-versa, or may involve arotation of an arbitrary number of degrees, e.g., to level the eye line,etc. (see also above).

An electrical system, software or firmware may be provided wherein animage of a portion of a photographic image is automatically produced.This may utilize a technique known as a “belief map” (see, e.g., USpatent application 2002/0114535 to Luo, incorporated by reference) todetermine the probability that a certain region within the principleimage is the main region of interest. Main subjects may be automaticallydetermined in an image (see, e.g., U.S. Pat. No. 6,282,317 to Luo etal., incorporated by reference). Regions of arbitrary shape and/or sizemay be extracted from a digital image. These regions may be grouped intolarger segments corresponding to physically coherent objects. Aprobabilistic reasoning engine for main-subject-detection may alsoestimate the region which is most likely to be the main subject of theimage. This technique may involve a set of feature extractions from theoriginal image which are then filtered through a tunable, extensible,probability network to generate the belief map. In this alternativeembodiment, the probabilistic “belief map” is generated by the mainsubject detection engine.

The above system of the alternative embodiment involving the generationof a belief map may generally involve some degree of computationalcomplexity. According to a preferred embodiment herein, informationgained from the detection and/or presence of faces in an image may beadvantageously used to determine a region or regions of interest withinan image, generally involving a reduction of computational complexityand making its application with resource-constrained portable orembedded systems very desirable.

A system, software and/or firmware may be provided that automaticallyrotates, crops and scales digital images for printing (see, e.g., U.S.Pat. No. 6,456,732 to Kimbell et al., incorporated by reference). Inthis embodiment, an optimal number of digital images may be fit onto asheet of paper of definite size in a printer. The system may or may notinvolve improving the images themselves, and/or may include one or morecomponents that serve to preserve the original image quality in theprinted form. In accordance with a preferred embodiment, the actions ofrotating, cropping and/or scaling an image may be based on criteriaother than those derived from the image as a whole, such as informationpertaining to faces in the image.

An embodiment involving automatic image cropping may use regionalintensity variations to separate areas of an image with uniformintensity levels from those with significant variation in intensitylevels (see, e.g., U.S. Pat. No. 5,978,519 to Bollman et al.). Aportrait, e.g., may be cropped from a uniform background, such as in theinstance of a passport photograph. In accordance with a preferredembodiment, however, a portrait may be extracted from a more complexbackground. Automatic cropping may be based on intensity and/or texturevariances of regions within an image. Face detection is preferably usedas an advantageous means to determine a region within an image forautomatic cropping.

In the context of automatic image rotation, and determining imageorientation, an embodiment including electrical, software and/orfirmware components that detect blue sky within images may be included(see, e.g., U.S. Pat. No. 6,504,951 to Luo et al., incorporated byreference) This feature allows image orientation to be determined oncethe blue-sky region(s) are located and analyzed in an image. Inaccordance with an alternative embodiment, other image aspects are alsoused in combination with blue sky detection and analysis, and inparticular the existence of facial regions in the image, to determinethe correct orientation of an image.

(iii) Color Adjustment and Tone Scaling of a Digital Image

A portion of an image may be modified in accordance with colorimetricparameters (see, e.g., US published patent application 2002/0105662 toPatton et al., incorporated by reference). Such image modification mayinvolve identifying a region representing skin tone in an image,displaying a plurality of renderings for the skin tone, selecting one ofthe renderings and/or modifying the skin tone regions in the images inaccordance with the rendering of the skin tone, e.g., as selected by theuser or automatically or semi-automatically selected using softwareand/or firmware. The skin tone information from a particular region ofthe image may be used to enhance the image. In accordance with apreferred embodiment, facial regions are detected within the image,based on which image enhancement is automatically or semi-automaticallyperformed.

In another embodiment, image color may be compensated when adjusting thecontrast of a digital color image (see, e.g., U.S. Pat. No. 6,438,264 toGallagher et al.). This may include receiving a tone scale function,calculating a local slope of the tone scale function for each pixel of adigital image, calculating a color saturation signal from the digitalcolor image, and/or adjusting the color saturation signal for each pixelof the color image based on the local tone scale slope. Imageenhancements may be applied to the entire image and/or may be based on aglobal tone scale function. In accordance with the preferred embodiment,such enhancement may be applied to a region of interest such as a facialregion. Characteristics of a region of the image may be used to applyautomatic enhancements to the entire image or, alternatively, the use ofwhole image characteristics or global characteristic functions may beused to apply automatic enhancements to selective regions of an image,such as facial regions.

A spatially blurred and/or sub-sampled version of an original image canbe used to obtain information regarding statistical characteristics of ascene or original image (see, e.g., U.S. Pat. No. 6,249,315 to Holm,incorporated by reference). This information may be combined with tonereproduction curves and other characteristics of an output device ormedia to provide an enhancement strategy for optimized output of adigital image. This processing can be performed automatically or bysimple, intuitive manual adjustment by a user.

(iv) Exposure Adjustment and Digital Fill Flash

A system, software, firmware or method for simulating fill flash indigital photography may be provided in accordance with preferred andalternative embodiments herein (see also US patent application2003/0052991 to Stavely et al.) A digital camera may be used to shoot aseries of photographs of a scene at various focal distances. Thesepictures may be subsequently analyzed to determine distances todifferent objects in the scene. Regions of these pictures may have theirbrightness selectively adjusted based on the aforementioned distancecalculations and may be then combined to form a single, photographicimage. In accordance with a preferred embodiment, information regardingthe existence of facial regions within the image is used, e.g., toparticularly selectively adjust the brightness of the facial regions.Moreover, automatic enhancement on a single image may be advantageouslyperformed in the preferred embodiment. Performing such enhancement on asingle image reduces the speed which a camera may otherwise need to becapture multiple images. Alternatively, several images may be combinedto form one. A multiplicity of images may be captured in thisalternative embodiment by a digital camera without the camera moving,generally involving a camera employing a very fast image captureprocess.

Another embodiment includes scene recognition method and a system usingbrightness and ranging mapping (see, e.g., US published patentapplication 2001/0031142 to Whiteside, incorporated by reference).Auto-ranging and/or brightness measurement may be used to adjust imageexposure to ensure that background and/or foreground objects arecorrectly illuminated in a digital image. Automatically adjustment ofimage exposure may be performed prior to image capture, or morepreferably after the image is captured.

In the preferred embodiment, corrections and enhancements of regionswithin digital images are performed preferably including entire faces orparts of faces that themselves form part of an overall image. This mayinclude a selected face or selected faces of multiple faces appearing inan image. For these preferred corrections and enhancements, fill flashis preferably used. Alternatively, image correction and enhancements maybe performed on entire digital images. This may involve correction ofimage exposure and tone scale (see, e.g., U.S. Pat. Nos. 6,473,199 toGilman et al. and 5,991,456 to Rahman et al., incorporated byreference).

Regional analysis and regional adjustment of image intensity or exposurelevels may be performed in accordance with preferred and alternativeembodiments. A method or apparatus may use area selective exposureadjustment (see, e.g., U.S. Pat. No. 5,818,975 to Goodwin et al.,incorporated by reference). A digital image can have the dynamic rangeof its scene brightness reduced to suit the available dynamic brightnessrange of an output device by separating the scene into two regions: onewith a high brightness range and one with a low brightness range. Abrightness transform may be derived for both regions to reduce thebrightness of the first region and to boost the brightness of the secondregion, recombining both regions to reform an enhanced version of theoriginal image for an output device.

In another embodiment, brightness adjustment of images uses digitalscene analysis (see, e.g., U.S. Pat. No. 5,724,456 to Boyack et al.). Animage may be partitioned into blocks and larger groups of blocks, thatmay be referred to as sectors. An average luminance block value may bedetermined. A difference may be determined between the maximum andminimum block values for one or more sectors. If this difference exceedsa pre-determined threshold, the sector may be marked active. A histogramof weighted counts of active sectors against average luminance sectorvalues may also be plotted and the histogram shifted to using apre-determined criteria so that the average luminance sector values ofinterest will fall within a destination window corresponding to a tonalreproduction capability of a destination application or output device.In accordance with a preferred embodiment, regions within an image arepreferably used, and even more preferably the presence or knowledge of ahuman facial region or facial regions within the image are used todetermine and/or apply image enhancement and/or correction to theregions or to the image as whole.

(v) Brightness Adjustment; Color Space Matching; Auto-Gamma

Further preferred and alternative embodiments involving face detectionand image enhancement may include brightness adjustment and colormatching between color spaces. For example, image data may betransformed from device dependent color spaces to device-independent labcolor spaces and back again (see, e.g., U.S. Pat. No. 6,459,436 toKumada et al., incorporated by reference). Image data may be initiallycaptured in a color space representation which is dependent on the inputdevice, and may be subsequently converted into a device independentcolor space. Gamut mapping (hue restoration) may be performed in thedevice independent color space and the image data may then be mappedback to a second device-dependent color space and sent to an outputdevice.

A system, software and/or firmware may be provided to correct luminanceand chrominance data in digital color images (see, e.g., U.S. Pat. No.6,268,939 to Klassen et al., incorporated by reference) In thisembodiment, transformations may be optimized between device dependentand device independent color spaces by applying sub-sampling of theluminance and chrominance data. In another embodiment, qualityimprovement of a printed image may be performed by automaticallydetermining an image gamma and then adjusting the gamma of a printer tocorrespond to that of the image (see, e.g., U.S. Pat. No. 6,192,149 toEschback et al., incorporated by reference). The printed quality of adigital image may be thereby enhanced, and preferably a digital imageitself may also be enhanced in this way. In an alternative embodiment,software or firmware provides for automatically determining the gamma ofa digital image. This information may be advantageously used to directlyadjust image gamma, or used as a basis for applying other enhancementsto the original digital image, such as face detection-based imageenhancement according to a preferred embodiment. In addition, agradation correction to an RGB image signal may be implemented allowingimage brightness to be adjusted without affecting image hue andsaturation (see, e.g., U.S. Pat. No. 6,101,271 to Yamashita et al.).

(vii) In-Camera Image Enhancement

In further preferred and alternative embodiments, improvements to adigital image may be suggested to a user after the image has beenacquired or captured by a camera (see, e.g., U.S. Pat. No. 6,516,154 toParulski et al., incorporated by reference). According to theseembodiments, a user may crop, re-size and/or adjust color balance eitherbefore saving a picture or before deciding to re-save or delete thepicture. The user may choose to re-take a picture using differentsettings on the camera. Suggestion for improvements may be made by thecamera user-interface.

(viii) Face-Based Image Enhancement

In preferred embodiments herein, automatically or semi-automaticallyimproving the appearance of faces in images based on automaticallyand/or manually detecting such facial images in the digital image is anadvantageous feature (see also US published patent application20020172419, to Lin et al., incorporated by reference). Lightnesscontrast and color level modification of an image may be performed toproduce better results. Moreover, using such information and otherinformation for detecting orientation, providing assistance as part ofan in-camera acquisition process, and/or providing assistance in acomposition or a slide show based on such information are features ofpreferred and alternative embodiments herein. The application mayautomatically detect the faces, or suggest portions of an image that maycorrespond to faces for confirmation and selection by a user, and thesoftware or firmware may allow the user to select additional imageportions that may correspond to faces not detected by the automatic orsemi-automatic processes of the software. Image enhancement according topreferred and alternative embodiment herein may be applied to a faceregion or face regions only, or the enhancement may be applied to theentire image, or selective and distinct corrections may be applied toboth background and foreground regions, particularly facial regions,based on knowledge of the presence of faces in the image and/or otherimage regions such as blue sky or other detectable features.

Auto-Focus

In accordance with a preferred embodiment, detection of a face or aprimary face among multiple faces in an image enhances the ability ofthe software and/or firmware of an image processing system, within acamera or external to a camera, is advantageously used to provide anaccurate focusing on the appropriate subjects. Distances to subjects maybe estimated, and automatic or manual determinations may be made as towhere the main subject matter exists.

A system, software and/or firmware may include an exposure measurementand/or focus detection feature that uses an image sensor such as photodiode array (MOS image sensor), or CCD (charge coupled device) (see,e.g., U.S. Pat. Nos. 4,047,187, and RE31370, to Mashimo et al.,incorporated by reference). In general with respect to preferred andalternative embodiments, a photodetector array device may be used forthe detection of images for image processing, particularly preferablyincluding faces and face detection, or from which the image may beacquired and/or captured and stored for image processing. A system maydefines an active auto focus system in which a beam of modulated energyis projected towards a subject to be focused upon with the energyreflected therefrom directed towards the detector array. An auto focuscircuit for a video or still camera may also include passive systemsthat do not use infra red signal reflections for auto-focus (see, e.g.,U.S. Pat. Nos. 4,638,364 and RE33682, incorporated by reference). Suchsystems may be based on analysis of images either based on highfrequency information, or alternatively a local contrast calculation(see, e.g., U.S. Pat. No. 4,638,364 and Japanese patent JP 5,260,360 A2,incorporated by reference).

A subject of an image, such as a facial region, may be based on acenter-weighted formula. Alternatively, a camera may focus on objectsbased on the direction that the eye which is looking into the viewfinderis pointing (see, e.g., U.S. Pat. No. 5,291,234, incorporated byreference). Such alternative system may generally further includespecial additional hardware in the camera viewfinder. The system of thepreferred embodiment which detects faces and generally utilized softwareand/or firmware to determine auto-focus based on size and/or location ofone or more primary faces is advantageous over this alternativehardware-based auto-focus system in that eye movement may involve notonly a fixation where the photographer holds his/her gaze at astationary point, but may involve saccade where the photographer moveshis/her gaze quickly between a few points. Also according to a preferredembodiment, electronic detection, acquisition and/or capture circuitry,software and firmware serves to detect and/or correct or enhance a focusof an image, preferably automatically or alternativelysemi-automatically as providing a choice or choices to a user to selectfrom, based on an analysis of the content of the image, and inparticular based on the location of faces in the image which may bedeemed to have primary importance among various regions on the image.

Selecting Regions of Interest

In further embodiments, various schemes may be used for selecting anarea or areas of interest from an electronically captured image, mostpreferably areas including faces or facial regions (see also UK patentapplication number GB0031423.7 entitled “automatic cropping ofelectronic images”, incorporated by reference). Regions of interest maybe automatically or semi-automatically selected within an image inresponse to a selection signal (see, e.g., US published patentapplication 2003/0025812, incorporated by reference). Panning across animage may be performed preferably keeping a selected region such as aselected face or faces in view. An image processing system may bepreferably employed with the panning method. A video camera system ofanother embodiment may implement tracking and/or zooming in relation toan object targeted by the camera (see, e.g., U.S. Pat. No. 5,812,193,incorporated by reference) For example, a user may teach the camera bypresenting the camera with a view of an object, such that the camera maythen seek to control a tracking motor so as to keep the object in view,and/or a zoom motor such that the size of the object with respect to theoverall image remains fixed at the region learned by the camera. In afurther embodiment, a model of a person's head may be provided such thatthe camera can correctly identify the head, or others like it, withinit's field of view. Thus the device seeks to maintain a lock on atarget. Such may be performed mechanistically or according to softwareand/or firmware provided within the camera. Advantageously, multipletargets such as multiple faces or facial regions may be tracked and/orzoomed, and preferably digitally enhanced, e.g., in view of aestheticconsiderations.

While an exemplary drawings and specific embodiments of the presentinvention have been described and illustrated, it is to be understoodthat that the scope of the present invention is not to be limited to theparticular embodiments discussed. Thus, the embodiments shall beregarded as illustrative rather than restrictive, and it should beunderstood that variations may be made in those embodiments by workersskilled in the arts without departing from the scope of the presentinvention as set forth in the claims that follow and their structuraland functional equivalents.

In addition, in methods that may be performed according to the claimsbelow and/or preferred embodiments herein, the operations have beendescribed in selected typographical sequences. However, the sequenceshave been selected and so ordered for typographical convenience and arenot intended to imply any particular order for performing theoperations, unless a particular ordering is expressly provided orunderstood by those skilled in the art as being necessary.

1. A method of digital image processing using face detection to achievedesired contrast and color balance of two or more detected faces,comprising: identifying two or more groups of pixels that correspond totwo or more faces within a digital image; identifying one or moresub-groups of pixels that correspond to one or more facial features ofeach of the two or more faces; determining initial values of contrastand color balance of pixels of the two or more groups of pixels based onvalues of said one or more sub-groups of pixels of said each of the twoor more faces, including calculating luminance and chrominancecharacteristics of said groups; determining initial contrast and colorbalance of the two or more faces within the digital image based on theinitial values; and determining adjusted values of pixels within thedigital image for adjusting the contrast or color balance, or both,respectively to the desired contrast or color balance.
 2. The method ofclaim 1, further comprising determining which of the sub-group of pixelsbelong to which said group of face pixels.
 3. The method of claim 1,said determining initial values of contrast or color balance, or both,of pixels being calculated based on the luminance or chrominancecharacteristics, or both, of said one or more sub-groups that correspondto one or more facial features.
 4. The method of claim 1, the one ormore facial features including an eye, two eyes, a mouth, lips, hair orhairline, ears, nostrils, nose bridge, eyebrows, chin, neck or a nose,or combinations thereof.
 5. The method of claim 1, wherein the one ormore facial features include at least two features, and the methodfurther comprises determining relative luminance and chrominance of theat least two features based on the initial values, and the initialvalues of contrast and color balance being determined based on therelative luminance and chrominance of the at least two features.
 6. Themethod of claim 1, further comprising: determining initial values ofrelative positions, orientations and pose angles of pixels of the two ormore groups of pixels based on values of said one or more sub-groups ofpixels of said each of the two or more faces, including calculatingspatial orientations of axes of ellipses fit to said groups; determininginitial relative positions, orientations and pose angles of the two ormore faces within the digital image based on the initial values; anddetermining adjusted values of pixels within the digital image foradjusting the relative positions, orientations or pose angles, orcombinations thereof, respectively, to the desired relative positions,orientations or pose angles.
 7. A portable image acquisition andprocessing device, comprising a processor, image sensor and lens toacquire digital images, and a memory having code embedded therein forprogramming the processor to perform a method of digital imageprocessing using face detection to achieve desired contrast and colorbalance of two or more detected faces, wherein the method comprises:identifying two or more groups of pixels that correspond to two or morefaces within a digital image; identifying one or more sub-groups ofpixels that correspond to one or more facial features of each of the twoor more faces; determining initial values of contrast and color balanceof pixels of the two or more groups of pixels based on values of saidone or more sub-groups of pixels of said each of the two or more faces,including calculating luminance and chrominance characteristics of saidgroups; determining initial contrast and color balance of the two ormore faces within the digital image based on the initial values; anddetermining adjusted values of pixels within the digital image foradjusting the contrast or color balance, or both, respectively to thedesired contrast or color balance.
 8. The device of claim 7, wherein themethod further comprises determining which of the sub-group of pixelsbelong to which said group of face pixels.
 9. The device of claim 7,wherein said determining initial values of contrast or color balance, orboth, of pixels includes calculating based on the luminance orchrominance characteristics, or both, of said one or more sub-groupsthat correspond to one or more facial features.
 10. The device of claim7, the one or more facial features including an eye, two eyes, a mouth,lips, hair or hairline, ears, nostrils, nose bridge, eyebrows, chin,neck or a nose, or combinations thereof.
 11. The device of claim 7,wherein the one or more facial features include at least two features,and the method further comprises determining relative luminance andchrominance of the at least two features based on the initial values,and the initial values of contrast and color balance being determinedbased on the relative luminance and chrominance of the at least twofeatures.
 12. The device of claim 7, wherein the method furthercomprises: determining initial values of relative positions,orientations and pose angles of pixels of the two or more groups ofpixels based on values of said one or more sub-groups of pixels of saideach of the two or more faces, including calculating spatialorientations of axes of ellipses fit to said groups; determining initialrelative positions, orientations and pose angles of the two or morefaces within the digital image based on the initial values; anddetermining adjusted values of pixels within the digital image foradjusting the relative positions, orientations or pose angles, orcombinations thereof, respectively, to the desired relative positions,orientations or pose angles.
 13. One or more non-transitoryprocessor-readable media having code embedded therein for programmingthe processor to perform a method of digital image processing using facedetection to achieve desired contrast and color balance of two or moredetected faces, wherein the method comprises: identifying two or moregroups of pixels that correspond to two or more faces within a digitalimage; identifying one or more sub-groups of pixels that correspond toone or more facial features of each of the two or more faces;determining initial values of contrast and color balance of pixels ofthe two or more groups of pixels based on values of said one or moresub-groups of pixels of said each of the two or more faces, includingcalculating luminance and chrominance characteristics of said groups;determining initial contrast and color balance of the two or more faceswithin the digital image based on the initial values; and determiningadjusted values of pixels within the digital image for adjusting thecontrast or color balance, or both, respectively to the desired contrastor color balance.
 14. The one or more processor-readable media of claim13, wherein the method further comprises determining which of thesub-group of pixels belong to which said group of face pixels.
 15. Theone or more processor-readable media of claim 13, wherein saiddetermining initial values of contrast or color balance, or both, ofpixels includes calculating based on the luminance or chrominancecharacteristics, or both, of said one or more sub-groups that correspondto one or more facial features.
 16. The one or more processor-readablemedia of claim 13, the one or more facial features including an eye, twoeyes, a mouth, lips, hair or hairline, ears, nostrils, nose bridge,eyebrows, chin, neck or a nose, or combinations thereof.
 17. The one ormore processor-readable media of claim 13, wherein the one or morefacial features include at least two features, and the method furthercomprises determining relative luminance and chrominance of the at leasttwo features based on the initial values, and the initial values ofcontrast and color balance being determined based on the relativeluminance and chrominance of the at least two features.
 18. The one ormore processor-readable media of claim 13, wherein the method furthercomprises: determining initial values of relative positions,orientations and pose angles of pixels of the two or more groups ofpixels based on values of said one or more sub-groups of pixels of saideach of the two or more faces, including calculating spatialorientations of axes of ellipses fit to said groups; determining initialrelative positions, orientations and pose angles of the two or morefaces within the digital image based on the initial values; anddetermining adjusted values of pixels within the digital image foradjusting the relative positions, orientations or pose angles, orcombinations thereof, respectively, to the desired relative positions,orientations or pose angles.